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Abstract Body Limit 4 pages single-spaced. 


Background / Context: Description of prior research and its intellectual context. 

Many researchers, policymakers, and university administrators have called for more 
students to major in Science, Technology, Engineering and Mathematics (STEM) fields to 
address the critical need for a diverse and technically skilled workforce that is prepared to 
address national problems and educate its future STEM leaders. One argument for increasing the 
number of students graduating with STEM degrees is that by increasing the number of students 
with these degrees that we can increase economic growth. One of the questions facing the nation 
is how best to increase the number of students obtaining degrees in STEM fields. The University 
of Maryland, Baltimore County (UMBC) received support from the National Science Eoundation 
(NSE) to comprehensively investigate cost effective and scalable strategies to increase the 
number of students graduating with STEM degrees. UMBC is known for its successful 
scholarship programs and for increasing the number of students who successfully complete 
STEM degrees and pursue graduate and professional degrees. UMBC’s Meyerhoff Scholarship 
Program has been recognized as a national model for increasing diversity among future leaders 
in science and engineering. The programs, as currently administered, can impact a limited 
number of students due the cost. Also, it is unclear as to what individual facets of these 
programs matter most for student retention and graduation. 

Purpose / Objective / Research Question / Focus of Study: Description of the focus of the research. 

The project’s primary purpose is to identify techniques that significantly impact retention 
and graduation rates among students who intend to major in STEM disciplines, to verify the 
estimates of these impacts through careful analysis, and to assess all costs relative to the 
effectiveness of these interventions. UMBC’s goal in the identification and application of 
techniques that have both high impact and low net costs will help us to better support its students 
as the project expands best practices that broaden participation and increase STEM graduation 
and retention rates. By conducting random assignment of students to interventions, we can 
evaluate the causal impacts of the interventions. Another component of the study is to evaluate 
the cost-effectiveness of these interventions. While it may be useful to know for example that 
faculty mentoring is critically important to student success, it is also important to know how 
costly an intervention would be to scale to all students who enter UMBC. Part of our analysis 
focuses on the cost-effectiveness of these interventions so as to guide university administrators 
on how best to implement programs that can impact the most students at the lowest cost. 

Setting: Description of the research location. (May not be applicable for Methods submissions) 

The research location is the University of Maryland, Baltimore County, 1000 Hilltop 
Circle, Baltimore, Maryland 21250. 

Population / Participants / Subjects: Description of the participants in the study: who, how many, key 
features, or characteristics. (May not be applicable for Methods submissions) 

Entering freshmen who were new to UMBC and who planned to major in STEM 
disciplines at UMBC were mailed personalized invitations by the Provost on his letterhead to 
participate in the iCubed@UMBC project. To participate in this research study, students must 
receive math placement exam scores that make them eligible for college algebra or higher-level 
mathematics courses and they may not be participants in one of UMBC’s scholarship programs. 
Current records identify 304 students in Cohort 1, 514 students in Cohort 2, and about 450 
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students in Cohort 3. Figures 1 and 2 in the appendix detail how many students were assigned to 
the five intervention groups for Cohort 1 and 2. 

Intervention / Program / Practice: Description of the intervention, program, or practice, including details of 
administration and duration. (May not be applicable for Methods submissions) 

Eligible students who signed the eonsent forms to partieipate were randomly assigned 
into five separate groups: 1) high-status faeulty mentoring with ongoing risk assessment, 2) 
professional staff mentoring with ongoing risk assessment, 3) plaeement in an aetive learning 
diseussion seetion in pre-ealeulus or caleulus elasses in the CNMS Aetive Seienee Teaehing and 
Learning Environment (CASTLE), 4) assistanee in forming and maintaining study groups, and 5) 
the eontrol group that offers a $50 UMBC Bookstore gift eard. Partieipation takes plaee over the 
fall and spring semesters with Cohort 1 held during the 2011-2012 AY, Cohort 2 held 2012-2013 
AY, and Cohort 3 held 2013-2104 AY 

Eaeulty Mentoring - Up to ten high-status, tenured faeulty mentor all students in this intervention 
by diseipline and reeeive $1,000 annual stipends regardless of the number of students mentored. 
Projeet protoeol with suggested meeting topies ealls for the faeulty mentors to hold two group 
meetings during the fall semester and one group meeting during the spring semester. The 
meetings foeus on sueeess in the diseipline and the iCubed@UMBC Program Coordinator assists 
the faeulty mentors in arranging meetings, eommunieating with students, and eondueting the 
ongoing risk assessment, whieh has five eomponents: 1) Seores from a first-day Quiz Zero given 
in key math elasses to assess preparedness; 2) Responses to an Early Warning Survey send to 
faeulty teaehing STEM eourses involving students in the mentoring interventions, 3) Responses 
to a eampus-wide Eirst Year Initiative (EYI) survey to faeulty monitor student progress, 4) 
Student self identifieation, and 5) Eall Semester grades. Students identified as being at risk are 
invited to sehedule an appointment with their faeulty mentors and mentors may use an 
established protoeol to help the students assess and address their risk faetors. 

Staff Mentoring - A professional staff advisor with STEM expertise mentors all students in this 
intervention, whieh follows a similar protoeol and risk assessment monitoring with the student 
meetings foeused on general sueeess in STEM majors. One researeh question is whether a high- 
profile faeulty mentor affeets students differently than a staff advisor. 

Aetive Learning - This intervention requires that partieipants sehedule a pre-ealeulus or ealeulus 
diseussion seetion in a speeifie eampus aetive learning elassroom (CASTLE), as is possible, and 
that other iCubed@UMBC partieipants do not sehedule a pre-ealeulus or ealeulus discussion 
section in this location. Our analysis of student records found that one of the largest determinants 
of whether a student stays in a STEM major is their grade in their first math class. This treatment 
is designed to allow us to evaluate whether the method in which students learn mathematics 
affects student retention. 

Study Groups - The iCubed@UMBC Program Coordinator follows a protocol to encourage and 
assist students in this intervention to form and/or participate in study groups. Group meetings 
are held at the beginning of the fall and spring semesters to introduce student-friendly 
information about the benefits of study groups. Eollow up, including office hours around 
campus, throughout the semester encourages study groups and address any issues related to study 
groups, as requested. Each semester ends with a pizza drawing to randomly reward study group 
formation on students who are currently enrolled in the same class as well as their schedules. 
Control Group - Students in this intervention receive no specialized services and are offered 
UMBC Bookstore $50 gift cards, which they sign for and present photo IDs. 
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All Groups - Students in all five interventions are eneouraged to take advantage of the many 
resourees and support serviees that are available to all students on eampus. They are asked to 
eomplete two eleetronie surveys with the first survey sent out at the beginning of the fall 
semester and the seeond survey sent at the end of the spring semester. The time to eomplete the 
survey is about 15 to 30 minutes per survey. Students are randomly invited to participate in 
Cohort-based focus groups to garner student opinions about the project and intervention. 

Significance / Novelty of study: 

Description of what is missing in previous work and the contribution the study makes. 

While many researchers recognize the value of random assignment, very few studies 
have implemented random assignment at the college level. The research team provide an analysis 
of four interventions that are known to affect student retention and graduation in STEM at 
UMBC using random assignment. In addition, other components of the study will: 1) analyze 
longitudinal data at UMBC to identify risk factors for STEM student success and 2) evaluate the 
cost-effectiveness of the identified interventions so as to inform the university and other 
universities about best practices in promoting student retention at lowest possible cost. 

Statistical, Measurement, or Econometric Model: 

Description of the proposed new methods or novel applications of existing methods. 

The research team leader randomly assigned participants in iCubed@UMBC to five 
different groups (four interventions and one control group). The research team is conducting an 
analysis of the effectiveness of these interventions on student success and will be looking at the 
effects of these interventions on first-year grades, student retention at the university and in the 
major as well as graduation from the university. When analyzing the effects of these 
interventions, we will be focusing on two particular effects. The first effect is the intent to treat. 
This is the fact that each person in the randomly assigned group was offered the services 
particular to that treatment. However, not every person will take up the services. The second 
effect is the treatment of the treated. These are students who actively participated in the 
intervention. In addition to administration records on students, we are also gathering survey data. 
Erom the survey data as well as records from the individuals running our interventions, we will 
be able to determine who actually used the interventions. 

Usefulness / Applicability of Method: 

Demonstration of the usefulness of the proposed methods using hypothetical or real data. 

The advantage of random assignment of students to interventions is that if assignment is 
truly random then it is not correlated with student characteristics. The project, therefore, no 
longer need to be concerned about self-selection. Individuals who are assigned to one 
intervention do not differ in measurable ways from students who are not assigned to the 
intervention. This allows for a clear analysis of the causal effects of the intervention. This was 
tested using observable characteristics and is presented in Table 1 and Table 2 in the appendix. 
The method is often difficult to implement, as individuals who are assigned to one treatment 
group may prefer to be in another. Another limitation of this method is that there are some 
concerns about providing beneficial resources to only students within a treatment. 

Research Design: Description of the research design. (May not be applicable for Methods submissions) 

The iCubed@UMBC research design is fairly straightforward given that the project uses 
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random assignment. The bigger issues have been 1) how best to reeruit students to the study 2) 
how to get reeruited students to partieipate in the interventions and complete the surveys, and 3) 
how to best handle different definitions of and varying competencies in active learning 
techniques among faculty and graduate students. The first Cohort that started in Fall 2011 has, in 
many ways, served the project well as a pilot activity. It has taken many steps and missteps to 
address these issues, especially in figuring out the best means to recruit students as well as how 
best to engage them in the interventions. At the suggestion of the members of our Internal and 
External Advisory Boards, project staff focused on building camaraderie among the students at 
the beginning of Cohort 2’s academic year in an efficient and effective way. With the help of the 
Office of Student Life, a protocol was created for one-hour team meetings during UMBC’s 
Welcome Week just before the start of classes. Team meetings were held in five large class 
locations at the same time with invitations from the Provost mailed to the recruited students’ 
homes. Each intervention was given a team name and color. Eree iCubed@UMBC tee shirts, in 
one of the five team colors, were distributed at the team meetings that included a brief overview 
explaining the purpose of the study and what was expected of the students. Emphasis was placed 
on the importance of each participant and the two surveys they would be asked to complete. 
Volunteers conducted activities intended to get students to interact and have fun. Volunteers 
filmed small group presentations by the students about iCubed@UMBC to be used in the 
creation of an iCubed@UMBC YouTube video. Guests, including the University President, the 
Provost, and the Eaculty Mentors, made special appearances at the meetings, which ended with 
ice cream treats for everyone. Participation increased among Cohort 2 and the protocol for 
Welcome Week team meetings was repeated for Cohort 3. 

Data Collection and Analysis: Description of the methods for collecting and analyzing data. 

(May not be applicable for Methods submissions) 

The iCubed@UMBC project would not be possible without the collaboration and 
cooperation between many divisions on campus. The largest resource for the project’s research 
team, in terms of data collection, has been the UMBC Office of Institutional Research, which 
provides comprehensive demographic information on all participants as well as additional 
information on their course-taking behavior, grades, and declared majors. 


Findings / Results: Description of the main findings with specific details. (May not be applicable for Methods 
submissions) 

Although the project continues to be in its data collection phase, the research team 
members hope to have an analysis of the effects of these interventions on first-year grades and 
second-year retention. The project is just beginning its third and final Cohort and the remaining 
time of the study will be devoted to data analysis and dissemination. 

Conclusions: Description of conclusions, recommendations, and limitations based on findings. 

The iCubed@UMBC experience, so far, has demonstrated the difficulties inherent in 
implementing a large-scale, random assignment study of entering freshman at a public four-year 
university and the necessity of involving much of the campus in supporting such an ambitious 
endeavor. Addressing these challenges has provided many useful lessons with broader 
applications to other initiatives. The planned analysis of data from over 1200 student 
participants in the three iCubed@UMBC cohorts promises a better understanding of key 
interventions that may increase student participation and success in STEM fields. 
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